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ABSTRACT 


The purpose of this study is to quantitatively assess 
Naval Academy graduates' perceptions of two aspects of their 
undergraduate education as engineering majors: 1) the 
extent to which their undergraduate education is relevant to 
their current profession, and 2) their level of preparedness 
as a result of their engineering education. The 
Accreditation Board of Engineering and Technology (ABET) 
identifies eleven 'student learning outcomes' that are 
utilized as the basis for assessing relevance and 
preparedness. Baseline data is established for engineering 
graduates of the Naval Academy between the years of 1985 - 
2005. In addition to the general analysis, graduates are 
grouped for comparison and analysis according to status 
(civilian and military), job type (technical and non¬ 
technical) and according to their particular undergraduate 
majors. The results indicate high levels of both 
applicability and preparedness for most of the eleven 
skills. Recommendations for future engineering program 
improvements are offered. 
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I. 


INTRODUCTION 


A. BACKGROUND 

The primary purpose of the United States Naval Academy 
is to provide graduates who then serve as junior officers in 
the U.S. Navy and Marine Corps. A significant emphasis of 
the development of midshipmen is on their undergraduate 
education. Graduates are awarded both a commission into the 
Navy or Marine Corps and a Bachelor of Science degree. On 
average, more than a third of each graduating class will 
have majored in one of eight different engineering fields. 

Following graduation, graduates serve for a minimum of 
five years on active duty in one of many different warfare 
communities within the Navy or Marine Corps. Many remain on 
active duty well past this minimum commitment, and others 
move on to careers within the civilian workforce. Regardless 
of where the Naval Academy graduates end up going, or what 
they end up doing, it is important to understand how their 
undergraduate education has impacted their careers since 
graduation. 

B. PURPOSE 

There are several purposes to this study. The primary 
purpose is to determine the extent to which Naval Academy 
graduates who majored in engineering believe their 
undergraduate education to be relevant to their current 
profession. This portion of the study uses the Accreditation 
Board of Engineering and Technology (ABET) 'student learning 
outcomes,' which are a set of 11 engineering related skills 
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or abilities (ABET, 2005, p. 1) as the basis for assessing 
relevance. The primary goals are: (a) to determine the 
extent to which Naval Academy graduates believe these skills 
are relevant to their current profession, and (b) to examine 
how well graduates believe they were prepared in these areas 
through their undergraduate education. 

Another purpose of the study is to establish a good 
body of baseline data about engineering majors who have 
graduated from the Naval Academy. Data are collected from 21 
graduating classes, between the years of 1985 and 2005. This 
is the first time that information (feedback) has been 
gathered from Naval Academy graduates that provides insight 
into how they view their undergraduate engineering 
education. With this in mind, the survey was designed to 
gather a broad base of data about what professions or jobs 
graduates are currently doing and how their undergraduate 
education has factored into their careers. 

The data received are rich for exploration and 
meaningful comparison. A portion of this study focuses on 
comparing graduates who are still in an active duty military 
status to graduates who have transitioned to the civilian 
work force, with regard to how they view their engineering 
education. Comparisons are also made between graduates in 
technical and non-technical careers, as well as across the 
eight different engineering majors available to Naval 
Academy students. 

C. RESEARCH QUESTIONS AND METHODOLOGY 

The primary research question is: Utilizing ABET's 
accreditation criteria for the Naval Academy's engineering 
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program as a means of measurement, to what extent do 
graduates believe that their undergraduate education in 
engineering is applicable to their professional lives, and 
how do they judge their level of preparation? Some secondary 
research areas entail comparisons between civilian and 
military respondents, as well as between technical and non¬ 
technical jobs. 

This is a quantitative study. As one of the purposes of 
the study is to develop broad - spectrum baseline data, some 
of the analysis is devoted to descriptive statistics that 
illustrate interesting trends or highlight areas that may be 
rich for meaningful comparisons. Most of the analysis uses 
comparisons between group means, conducted utilizing paired 
sample and independent sample t-tests and analysis of 
variance tests (ANOVA). These tests focus on respondents' 
answers to questions concerning the applicability of and 
preparation achieved through their undergraduate engineering 
education. Again, the civilian and military groups are 
compared; and contrasts are made across the engineering 
fields. 

D. BENEFITS OF THE STUDY 

There are several possible benefits to this study. The 

primary benefit is the establishment of baseline data about 

engineering graduates of the Naval Academy over the past 20 

years, which can be useful in several ways. It gives leaders 

at the Naval Academy information they can in turn provide to 

the Accreditation Board for Engineering and Technology, as 

one of the requirements for accreditation renewal. More 

importantly, it provides information that can be used as a 

tool for engineering program improvement at the Naval 
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Academy, by identifying program strengths and weaknesses. 
This study enables leaders at the Naval Academy to assess 
how well we are preparing graduates, from an educational 
perspective, for their professional lives in the Navy, 
Marine Corps, and beyond. 

E. ORGANIZATION OF THE STUDY 

The study is organized into five chapters. Chapter I 
is an introduction and a brief explanation of the study. 
Chapter II provides a literature review of information 
relevant to the history and background of ABET and the 
development of their 'learning outcomes,' an overview of 
approaches to program evaluation, the accreditation 
practices of other universities, and survey implementation 
practices. Chapter III provides and in depth look at the 
research methodology used, along with a description of the 
site of the study and the survey instrument that is 
utilized. Some descriptive statistics characterizing the 
sample are also presented in chapter three. Chapter IV is 
dedicated to data analysis. Chapter V provides the results 
of the study along with some discussion of the implications 
associated with the results. Chapter V also discusses some 
of the limitations of the study, and ends with 
recommendations for further research. Finally, the 
appendices include the survey instrument in its entirety, 
along with frequencies and descriptive statistics for the 
data set that will be useful for further research. 
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II. LITERATURE REVIEW 


A. INTRODUCTION 

This chapter begins with a detailed history of the 
Accreditation Board for Engineering and Technology (ABET) 
and describes the recent development of ABET's current 
accreditation standards, known as Engineering Criteria 2000 
(EC2000) . These criteria were developed in response to both 
the evolving demands of the engineering profession and 
advancements in the realm of evaluation. The discussion then 
turns to the specific aspect of EC2000 that is addressed in 
this study, namely Criterion 3, or Program Outcomes 
Assessment. After defining the Criterion 3 outcomes, this 
chapter addresses ways in which to choose outcome 
indicators. 

The chapter then addresses the concept of program 
evaluation, with specific emphasis on survey implementation 
and utilization as a tool to conduct a program evaluation. 
Additional methods of evaluation are briefly discussed, with 
emphasis on the evaluation of learning outcomes. The chapter 
continues with a discussion about accreditation practices at 
various universities and colleges, as well as design 
techniques for courses that satisfy the ABET accreditation 
criteria. The chapter concludes with a brief discussion of 
teaching ethics as part of engineering curricula. 

B. ABET HISTORY 

The Accreditation Board for Engineering and Technology 
(ABET) Web site provides historical background for the 
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organization (ABET, 2007). ABET (originally named the 
Engineers Council for Professional Development, or ECPD) was 
founded in New York in 1932, as a composite of seven 
engineering societies that covered all the major engineering 
fields. Erom the beginning, the organization focused on the 
guidance, training, education and recognition of both 
engineering students and academic institutions. The ECPD 
developed into an accreditation society almost immediately 
following its inception, and in 1980 adopted the official 
name ABET in order to reflect this emphasis on 
accreditation. 

ABET has grown to include 28 professional societies and 
accredits more than 2700 programs at more than 550 colleges 
and universities, including seven programs at the United 
States Naval Academy. These include Aerospace, Electrical, 
Mechanical and Systems Engineering, accredited since 1970, 
Naval Architecture and Ocean Engineering, accredited in 
1972, and Computer Science in 1987 (ABET, 2007) . 

Eollowing a lengthy development period, ABET adopted a 
revolutionary approach to accreditation with its 
implementation of Engineering Criteria 2000 (EC2000). 
Enacted in 1997, EC2000 has allowed ABET to shift its focus 
during accreditation procedures to "what is learned rather 
than what is taught" (ABET, 2007). 

C. DEVELOPMENT OF EC2000 

By the end of the 20^*^ century, the curricula of 
engineering education had evolved from a focus on 
engineering practice and application toward a focus on 
mathematics, engineering theory and applied science (Prados, 
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Peterson & Lattuca, 2005) . As a result, a gap began to 
develop between the skill demands of practicing engineers in 
the industrial workforce and the skills that engineering 
education was providing to students. Prados, et al (2005) 
explain that, with continuing trends toward globalization 
and advancements in the information technology area, many 
employers began to recognize that sound technical skills 
alone would not dictate success within the profession. In 
particular, employers reported the need for young engineers 
to gain proficiency in communication skills, quality 
assurance, leadership and participation in teams and work 
groups, commitment to continuous education and learning, 
innovative thinking and creativity. 

Until the development of EC2000, ABET's accreditation 
criteria focused on quantitative measurement of engineering 
program inputs, such as number of faculty members or seat 
time (number of hours per student) in a particular subject. 
These criteria, while relevant and easy to measure and apply 
universally, did not address some of the engineering skills 
and many of the general management skills associated with 
success in the profession. ABET evaluators spent their time 
number crunching and auditing programs, rather than 
providing qualitative professional assessment. Not until the 
early 1990's did ABET realize that their strict adherence to 
measurement of program inputs had made them partially 
responsible for the widening rift between the needs of the 
engineering practice and the outcomes of engineering 
educational programs. After a substantial review process, 
ABET released the new EC2000 (Engineering Criteria 2000), 
which shifted emphasis away from standardized compliance to 
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program input requirements and toward and emphasis on 
defining program objectives and learning outcomes (Prados, 
Peterson & Lattuca, 2005) . 

EC2000 was piloted at five different institutions in 
1996 and 1997. Following the pilot run, EC2000 became 
optional for the three-year period from 1998 - 2000. During 
this time, institutions could volunteer to be evaluated 
using the new criteria, or continue to use the traditional 
ABET criteria. The number of institutions that chose to 
utilize the EC2000 criteria increased from 21% in 1998 to 
83% in 2000(Prados, Peterson & Lattuca, 2005) . 

D. PROGRAM OUTCOMES 

EC2000 divides the criteria for program accreditation 
into eight different areas that are listed in order below, 
as found on ABET's official Website (ABET Commission, 2006, 
pp. 1-3) : 

Criterion 1: Students 

Criterion 2: Program Educational Objectives 

Criterion 3: Program Outcomes Assessment 

Criterion 4: Professional Component 

Criterion 5: Faculty 

Criterion 6: Facilities 

Criterion 7: Institutional Support and Financial 

Resources 

Criterion 8: Program Criteria 

Feedback from graduates is a vital component of an 
engineering program review when attempting to satisfy 
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Criterion 3, "Program Outcomes Assessment." According to 
Criterion 3, ABET requires that an institution be able to 
produce, assess and document the achievement of designated 
"skills, knowledge and behaviors" (ABET, 2005, p. 1) that a 
student should achieve by the time of graduation. These 
skills, knowledge and behaviors are further categorized as 
items 'a' through 'k,' listed below in Table 1: 


Table 1: Criterion 3 Program Outcomes 


a 

an ability to apply knowledge of mathematics, science and 

engineering 

b 

an ability to design and conduct experiments, as well as to 
analyze and interpret data 

c 

an ability to design a system, component, or process to meet 
desired needs within realistic constraints such as economic, 
environmental, social, political, ethical, health and safety, 
manufacturability, and sustainability 

d 

an ability to function on multi-disciplinary teams 

e 

an ability to identify, formulate and solve engineering problems 

f 

an understanding of professional and ethical responsibility 

g 

an ability to communicate effectively 

h 

the broad education necessary to understand the impact of 
engineering solutions in a global, economic, environmental and 
societal context 

i 

recognition of the need for, and ability to engage in life-long 
learning 

j 

a knowledge of contemporary issues 

k 

an ability to use the techniques, skills, and modern engineering 
tools necessary for engineering practice. 


An important aspect of ABET's EC2000 is that it shifts 

focus from an emphasis on program inputs to a focus on 

program outcomes. Eor our purposes, it may be useful to 

define program inputs, processes, outputs and outcomes, and 

9 






emphasize the difference between outputs and outcomes. 
Rogers (2000) provides useful definitions of these terms. 
Inputs are those things that are brought into an engineering 
program by students (test scores, credentials, 
competencies), faculty (credentials, experience, values) and 
the institution itself (facilities, equipment, resources). 
Processes focus on what the institution does to exert 
influence on and work with inputs. These include student 
choice of major and courses, faculty teaching load and class 
size, and institutional procedures and governance. Outputs 
can be seen as the quantitative, concrete results of the 
processing of inputs. These include grades, retention and 
employment statistics. Outcomes, on the other hand, are less 
tangible. They are the knowledge, skills, attitudes, values 
and behaviors that a student develops as the result of 
having gone through the educational program (Rogers, 2000) . 

Rogers (2000) also illustrates the increased level of 
difficulty when an institution moves from classroom 
assessment to program assessment. The problems lie with the 
degree of complexity, time span, level of specificity, 
accountability, level of faculty buy-in and cost (Rogers, 
2000). Evaluating an entire engineering program is more 
complex than evaluating a course, because the desired 
outcomes are combined across the entire spectrum of the 
curriculum. It takes much more time to evaluate a program, 
because a student must be allowed to complete the program in 
its entirety. Program assessment is less specific in nature, 
and it is hard to assign accountability for success or 
failure to individuals within the program. Finally, program 
assessment is much more costly in terms of time, money and 
resources . 
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E. CHOOSING OUTCOME INDICATORS 

Periodically, the U.S. Naval Academy's Engineering 
department must "renew" the accreditation of its entire 
program. The process is run by the Accreditation Board of 
Engineering and Technology (ABET); it requires a thorough 
review of the program from many different aspects, one of 
which is an assessment of ABET's "student learning 
outcomes." While there are several ways to evaluate these 
outcomes, the evaluation method most relevant to this study 
is feedback from graduates. In their responses to a survey 
distributed in the fall of 2005, graduates had the 
opportunity to assess the value of their undergraduate 
education. 

Prior to 2005, there had never been an external 
evaluation of the U.S. Naval Academy's Engineering program 
analyzing feedback from graduates. With ABET's new "student 
learning outcomes" as a basis for this study, the intention 
is to determine to what extent graduates believe these 
outcomes are or are not applicable to their professional 
lives and to what degree these learning outcomes were 
successfully achieved during their four years at the 
Academy. 

ABET is the recognized accreditation source for all of 
the Engineering programs in the country (Brizendine, 2004). 
It is a federation of 32 professional and technical 
societies that has been functioning for more than 70 years. 
Every six years, an Engineering program that desires 
accreditation is required to satisfy ABET's criteria and 
submit the results to an ABET Board of Visitors for 


approval. Brizendine (2004) tells us that, historically, the 
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accreditation has been based primarily on the capabilities 
and the capacities of the program itself. In 1996, however, 
ABET introduced an entirely new method (in addition to, 
rather than in place of those already in use) to evaluate 
the quality of a given Engineering program. This new method 
requires an evaluation of program outcomes (Eelder, Brent, 
2003) . 

ABET's guidelines for measuring outcomes are based on 
their list of eleven "student learning outcomes" (identified 
as 'a' through 'k, ' see Table 1), which were implemented as 
part of their Technology Criteria 2000 (Brizendine, 2004). 
These outcomes are subject to a good deal of interpretation 
by the Engineering departments of the various institutions, 
but McGourty, Besterfield-Sacre and Shuman (1998) attempt to 
provide a way to categorize them. They divide these desired 
outcomes into three types: cognitive (what we "know"), 
behavioral (what we "do"), and attitudinal (what we "think" 
or "feel"). The authors believe these outcomes to be 
measurable, but they reason that there is no score that can 
be applied universally to determine passing or failing. 
McGourty, et al. (1998) believe that the advantage to vague 
wording of the "learning outcomes" is that it leaves it up 
to the individual Engineering departments to best define 
them (and evaluate them) in order to align the outcomes with 
their unique program needs. 

Aft (2002) explains that there are several different 
ways to evaluate the "student learning outcomes." An 
Engineering program can conduct an internal self-study, 
which relies on inputs and feedback from faculty and current 
students to assess the program. Equally important is the 
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concept of external assessment methods, which could include 
the periodic ABET visit or the survey of graduates to 
evaluate the program. Aft (2002) reasons that a quality 
program will utilize a variety of methods for both internal 
and external assessment. 

There has been research conducted on how to best 
implement and evaluate the "student learning outcomes" on an 
internal level. Specifically, the goal has been to take 
feedback from faculty, staff and students enrolled in the 
program and utilize that information to effect positive 
change. Felder and Brent (2003) tell us that this results in 
"change from within" in terms of clarified learning 
objectives, and improvements in instruction and assessment. 
There have also been studies conducted on successful methods 
for externally evaluating an Engineering program, as will be 
discussed later in this chapter. 

Scales, Owen, Shiohare and Leonard (1998) assert that 
evaluation begins when the faculty and staff of a program 
identify the desired outcomes of a course or program, and 
then identify which indicators will be used to measure these 
outcomes. Indicators can be classified in different ways, 
depending on what is to be measured (i.e., attitudes, 
behaviors, knowledge, performance) and how it is to be 
measured (qualitative vs. quantitative), or even by who is 
doing the measuring (i.e., exit surveys given by the 
institution, universal EE exams given nationally) (Scales, 
Owen, Shiohare & Leonard, 1998) . Availability of time and 
financial resources may be the biggest determinant when 
selecting an indicator. 
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Scales et al. (1998) identify seven different types of 

outcome indicators, one of which is alumni surveys. Alumni 
surveys are advantageous because they are relatively 
inexpensive to administer, results can be analyzed quickly 
(if quantitative data are collected) , and they allow an 
institution a way to maintain contact with its graduates. 
Alumni can provide useful biographical data and feedback 
concerning the quality and content of their education, and 
an institution can track how these evaluations change over 
time (Scales et al., 1998). 

Unfortunately, there are also several drawbacks when 
using alumni surveys. They suffer from low response rates 
and are not ideal for addressing complex issues, as there is 
no means to answer respondents' questions if there is an 
unclear or misunderstood survey question. Similarly, there 
is no way to guard against untruthful or misleading 
responses. Thus, alumni surveys are most effective when 
designed as brief, simple questionnaires (Scales et al., 
1998). Overall, Scales et al. (1998) report that in a survey 

given to the representatives of engineering programs 
attending the 1997 Best Assessment Processes in Engineering 
Education: A Working Symposium conference, the results 

indicated that alumni surveys were both highly utilized and 
believed to be very useful. 

F. EVALUATION 

Patton (1987) offers an excellent distinction of the 
differences between formative and summative evaluations. 
Formative evaluations are conducted in order to allow for 
program improvement, whereas summative evaluations are 


utilized to determine 


overall program effectiveness 
14 



specifically useful when making a determination as to 
whether a program should continue or not (Patton, 1987). 
When a program is looking to improve quality, a formative 
evaluation should be utilized. This allows a program to 
identify strengths (what is working well) and weaknesses 
(what processes are in need of improvement). Also central to 
the idea of formative assessment is the concept of feedback 
not only must feedback be continuously given by the 
evaluator, but that feedback must also be acted upon in such 
a manner as to improve program quality. In other words, the 
processes of evaluation and providing feedback themselves 
become an integral part of continuous improvement of program 
quality (Patton, 1987). 

Patton (1982) further clarifies the distinction between 
formative and summative evaluation by pointing out that 
summative evaluation results in a judgment being made about 
the composite worth or effectiveness of a program. Summative 
evaluations, in general, tend to focus on program outcomes, 
and formative evaluations tend to focus on program 
processes. Patton is quick to point out that there may be 
some blurring of the lines when deciding exactly what to 
focus on (process or outcome?), and that the important 
distinction occurs when deciding on the purpose of the 
evaluation (Patton, 1982). 

Royse, Thyer, Padgett and Logan (2001) discuss the fact 
that, although many experts believe formative evaluation to 
be synonymous with process evaluation, this is not 
necessarily true. The key difference is that a formative 
evaluation is typically conducted early in the development 
of a program, whereas a process evaluation can be conducted 
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at any time, even when the program has ended (Royse et al., 
2001). Again, it is important to note that the intention of 
the evaluation is still program improvement, and to 
differentiate between process and formative evaluation is 
simply a question of when the evaluation occurs within the 
lifetime of the program. 

Patton (1987) describes the inherent tradeoffs that are 
inescapably present when conducting any type of evaluation. 
Tradeoffs are unavoidable, due to realistic challenges 
associated with working with constrained resources and time. 
One particularly relevant tradeoff scenario that Patton 
addresses is the question of depth versus breadth. Within 
the confines of constrained resources and time, an evaluator 
must at some point make decisions about how broad or how 
deep the evaluation will run. Exploratory studies, Patton 
says, lend themselves more toward a focus on breadth, which 
allows for a wide range of baseline data; a focus on depth 
and detail may be more desirable when evaluating specific 
experiences or outcomes (Patton, 1987). 

Another key consideration in evaluation is units of 
analysis to be studied. On the topic, Patton (1987) states 
that "the key factor in selecting and making decisions about 
the appropriate unit of analysis is to decide what unit it 
is that you want to be able to say something about at the 
end of the evaluation" (p. 51) . Thus, making choices about 
which units of analysis will be used during an evaluation 
can also help to answer questions concerning breadth versus 
depth. 

The process of obtaining a target sample is of great 
importance in evaluation. Of many different sampling 
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methods, Patton (1987) provides some insight into one that 
is of particular importance to this study - criterion 
sampling. Criterion sampling occurs when all chosen cases 
"meet some predetermined criterion of importance" (Patton, 
1987, p. 56) . For example, identifying all graduates of a 
university who majored in an engineering field would 
constitute criterion sampling. The purpose, Patton (1987) 
tells us, is "to be sure to understand cases which are 
likely to be rich in information," because their information 
will likely lead evaluators toward program improvement (p. 
56) . 

Patton (1982) also highlights the importance of 
determining specifically what information is required prior 
to implementing a survey or questionnaire. He discusses four 
different types of questions that can be asked, based on 
what an evaluator wishes to find out: (a)behavior questions 
inquire about actions (what people do) ; (b) opinion 

questions inquire about thoughts (what people think); (c) 

feeling questions inquire about emotions (what people feel); 
and (d) knowledge questions inquire about facts (what people 
know) (1982) . For purposes of this study, the focus is 
primarily on the thoughts and opinions of survey 
respondents . 

G. BEST PRACTICES FOR SURVEY IMPLEMENTATION 

Couper, Traugott and Lamias (2001) conducted a study at 

the University of Michigan to identify some traits of Web - 

based surveys that can either add to or detract from their 

effectiveness with respect to response rates and data 

quality. The authors recognize the growing trend of Web - 

based survey utilization, and they identify some key 
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advantages, such as a decrease in turnaround time between 
questionnaire delivery and response and ease of access to a 
large population through the Web. The primary focus, 
however, is on the pros and cons of various types of Web 
survey design. 

In one experiment, the effect of including a progress 
indicator for the survey respondent was studied. While a 
larger percentage of respondents completed the survey if 
they were able to track their progress toward completion, 
the progress indicator also involved a more complex survey 
design which led to an increase in survey completion time 
(Couper et al., 2001). In another experiment, the effects of 
placing single versus multiple items on a page were studied. 
When a single item was placed on each page, the survey took 
longer to complete; when multiple items were placed on each 
page, response time decreased, but the correlation between 
answers increased (Couper et al., 2001) . The third 
experiment compared the response rates when respondents were 
given the option of either entering data in a text field or 
using the computer's mouse to click on a button. Response 
rates were significantly higher when respondents could use 
the mouse to enter all information, rather than entering 
data into a text field (Couper et al., 2001). 

These types of experiments seem to indicate that, when 
faced with the task of designing and implementing a Web - 
based survey, the designer will have to make choices based 
on what type of information they are attempting to retrieve. 
There will be trade-offs associated with each choice. 

Schwarz (1995) tells us that "respondents use features 
of the questionnaire to determine the meaning of a question 
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and to generate a useful answer" (p. 161) . Regardless of 
whether it is intended to be unbiased or neutral, there are 
certain aspects of a survey that force respondents to make 
interpretations and judgments. For example, the ordering of 
questions can have an impact or, in the case of a survey 
that uses a numerical rating scale, the words used to define 
the extreme high and low values on the scale can influence 
respondents, as can the numerical values chosen to represent 
different ratings (Schwarz, 1995) . For example, in one 
experiment conducted by Schwarz (1995), respondents to a 
survey were statistically much less likely to report a value 
below the midpoint on a rating scale if the scale ranged 
from -5 to 5 as opposed to a scale that ranged from 0 to 10. 

H. ACCREDITATION AT OTHER INSTITUTIONS 

McGourty, Sebastian and Swart (1998) describe the 
process that one institution, the New Jersey Institute of 
Technology (NJIT), implemented in order to achieve a 
comprehensive internal assessment of engineering education. 
The team at NJIT followed a five-step process to develop 
their program. The first step, defining objectives, 
strategies, and outcomes, required the team to identify the 
students and courses to which the assessment program would 
apply; then, faculty input (through focus groups and 
surveys) was used to decide which student learning outcomes 
would be assessed. 

The outcomes chosen align with the outcomes associated 
with ABET's EC-2000. The second step required the selection 
of assessment methods, with a focus on reliability. 


validity, fairness and, perhaps most importantly, choosing 

only those methods that could provide information useful to 
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continuous program improvement. Step three was a pilot run 
of the assessment process; step four was full implementation 
of the assessment process. The goal of step five was to 
close the loop on the process by applying the feedback from 
the assessment toward improvement of both the academic 
curriculum and the assessment process itself (McGourty, 
Sebastian & Swart, 1998). 

There were four assessment methods chosen by the NJIT 
team at the start of this program. The process began with a 
survey given to students prior to beginning a course. The 
students would then, with help from faculty, maintain a 
portfolio that tracked their progress through the courses 
chosen for assessment. A peer assessment tool (the "Student 
Developer") allowed students to give feedback to each other 
throughout the curriculum. Finally, at the end of the 
course, the faculty would observe and evaluate an oral 
presentation given by individual students, which would allow 
faculty to assess student outcomes for the course. Data 
useful in both assessing student learning and improving 
program composition were obtained from all methods with the 
exception of the portfolio, which was deemed to be too 
laborious for students and faculty and subsequently dropped 
(McGourty, Sebastian & Swart, 1998). 

Regan and Schmidt (1999) outline the evaluation process 
used by the University of Maryland, College Park, to 
determine if the "student learning outcomes" are achieved. 
Their evaluation method, which spanned the time period from 
entry into the program until five years after graduation, 
culminated with a survey of graduates to determine if the 
"student learning outcomes" were applicable in their 
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professional lives and to what extent those outcomes had 
been achieved during their undergraduate education. Their 
results provided them with a source of external feedback 
that proved to be beneficial when attempting to evaluate the 
achievement of ABET's student learning outcomes. 

I. OTHER METHODS OF EVALUATING LEARNING OUTCOMES 

In this section, different methods of assessing 
learning outcomes are discussed. The majority of the 
assessment methods were developed specifically with ABET's 
EC2000 criteria in mind, for use in determining the level to 
which Criterion Three (See Table 1) was attained. 

Christy and Lima (1998) advocate a method of assessing 
student learning that could possibly be useful as 
institutions attempt to measure EC2000 Criterion Three, or 
student learning outcomes. This method involves the use of 
portfolios maintained by the individual students throughout 
their time in the engineering program. One section of the 
portfolio would list course-specific desired goals or 
outcomes for students. Periodically, the students would 
evaluate their progress toward mastery of the outcomes, 
based on self - evaluation of their work in the course. 

The intention is to allow the student to take 
responsibility for their own education; with the help of the 
portfolio process of documentation and criterion - based 
self assessment to guide them toward the achievement of the 
desired outcomes. In a study conducted at Ohio State 
University, both freshmen and senior engineering students 
were required to maintain portfolios in one particular 
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class. 78% of the seniors and 80% of the freshmen reported 
that the portfolios did in fact enhance their learning 
experience (Christy & Lima, 1998). 

One relatively unexplored method of evaluating the more 
technical of ABET's learning outcomes is the nationally 
standardized Fundamentals of Engineering (EE) Exam (Watson, 
1998) . This eight - hour test covers 15 different areas of 
technical engineering science, and is administered to 
graduating engineering students who may wish to continue 
toward a professional engineering career. While this test 
could provide some rich data with which to assess student 
learning, it is rendered ineffective as a measurement tool 
by several factors. First, the National Council of Examiners 
for Engineers and Surveyors (NCEES) is bound by privacy and 
release restrictions. Second, even if NCEES were allowed to 
furnish test results, the test is taken on a predominantly 
voluntary basis. Many engineering students never take the 
exam, and those who do take it tend to be highly motivated 
(Watson, 1998) . 

J. DESIGN TECHNIQUES FOR COURSES THAT SATISFY ABET 

Shaeiwitz (1998) discusses the important role that 
classroom assessment can play in the development of a 
comprehensive program evaluation. In particular, the author 
discusses the importance of nested feedback loops, in which 
measurement of outcomes and feedback occur regularly 
throughout the students' progression through a curriculum. 
Using the concept of nested feedback loops, engineering 
programs can benefit from both formative and summative 
assessment. 
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In formative assessment, the audience is a particular 
instructor or faculty member, and the goal is to improve 
student learning at the course level (Shaeiwitz, 1998) . 
Formative assessments represent short-term feedback loops, 
where the time passage between measurement and feedback is 
minimized. 

With summative assessment, the goal is to evaluate the 
effectiveness of the educational curriculum taken as a 
whole, and the audience is external to the department. 
Summative assessment evaluates the entire educational 
process, from beginning to end. Shaeiwitz (1998) believes 
that a program designed to satisfy ABET's EC2000 criteria 
will take advantage of nested feedback loops, which provide 
multiple opportunities for formative assessment and program 
improvement. 

K. ETHICS IN ENGINEERING EDUCATION 

Haws (2001) discusses the importance of teaching ethics 
to engineers, as a way to help students to think 
divergently. By this, he implies that it is important to 
gain a broader perspective by looking, through the eyes of a 
non-engineer, at the potential impacts of decisions on a 
wide range of people and things. Teaching the Professional 
Code of Ethics solely by presenting the ethical 
considerations summarized in that document is insufficient 
(Haws, 2001). A good understanding of ethical considerations 
will most likely be achieved through a number of different 
approaches, including professional codes, humanist readings, 
theoretical grounding, ethical heuristics, case studies and 
service learning (actively applying ethics to community- 
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based projects). Haws (2001) recognizes value in all six of 
these methods, but favors a combination of theoretical 
grounding and service learning. 

L. SUMMARY 

This chapter has covered the history of the 
Accreditation Board for Engineering and Technology (ABET), 
highlighting the relatively recent changes that ABET has 
made to their accreditation criteria. The changes most 
relevant to this study involve the evaluation of ABET's 
Criterion Three 'Student Learning Outcomes.' Best practices 
for choosing and evaluating indicators of these outcomes 
were explored in detail. This chapter also explored the use 
of various methods of evaluation, with specific emphasis on 
surveys as tools for program evaluation. Einally, some 
background was given concerning accreditation at other 
institutions. 
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Ill.METHODOLOGY 


A. INTRODUCTION 

There are several purposes to this study. One goal is 
to establish baseline data about engineering graduates of 
the United States Naval Academy over the past 20 years. The 
bulk of the analysis is devoted to an overall assessment of 
the relevance of engineering education to Naval Academy 
graduates and the level of preparedness afforded them 
through their undergraduate education. 

Additionally, an attempt is made to explore the 
differences (if any) that exist between military and 
civilian graduates of the United States Naval Academy who 
majored in engineering, with respect to their level of 
preparedness for their profession and the level of 
applicability of their undergraduate education to their 
professional lives. Finally, comparisons are made between 
graduates in technical and non-technical career fields, as 
well as across the different engineering majors. 

The Accreditation Board for Engineering and Technology 
(ABET) Criterion Three 'Student Learning Outcomes' (See 
Table 1) provide the basis for all analyses and comparisons, 
and this chapter provides background information on how the 
study was conducted. The site of the study, survey 
instrumentation and implementation procedure, sample 
description and analysis procedures are all covered in 
detail. 
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B. SITE OF STUDY 

This study was conducted at the United States Naval 
Academy in Annapolis, MD. The Naval Academy is a four-year 
institution whose graduates are awarded both a Bachelor of 
Science degree and an officer's commission into the Navy or 
Marine Corps as either ensigns or second lieutenants. The 
minimum active duty service requirement following graduation 
is five years; many graduates remain on active duty past the 
minimum commitment, and many take their education and 
military experience and join the civilian workforce. 

Between the years 1990 - 2005, an average of 1246 men 
and women were admitted to the Naval Academy each year. 
During this same time span, an average of 964 men and women 
graduated and were commissioned into the Navy or Marine 
Corps each year. The average attrition rate for each 
incoming class was 22.5%, which means that an average of 
77.5% of all men and women admitted to the Naval Academy 
went on to graduate during this time period (U.S. Naval 
Academy Institutional Research, 2004). 

There are 19 different majors offered at the Naval 
Academy in areas of Engineering and Weapons, Mathematics and 
Science, and Humanities and Social Sciences. Selection of 
major is voluntary, and an average of 35% of each class of 
Midshipmen will choose to major in an engineering field. For 
this study, the focus is on graduates who studied in one of 
eight available engineering majors. In particular, this 
study deals only with engineering majors who graduated 
between the years of 1985 - 2005. During this 20 - year time 
span, a total of 7572 graduates of the Naval Academy majored 
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in some field of engineering. For purposes of this study, 
these 7572 graduates are the population. 

C. INSTRUMENTATION 

The survey instrument is essentially broken up into 
three distinct sections. The intent is to track information 
about the professional lives of respondents between the time 
of their graduation and present, then gather information 
about the applicability of their undergraduate education to 
their career and gauge the level of preparation each 
respondent feels was achieved through their undergraduate 
education. 

The survey (See Appendix A for the complete version) 
begins with 13 questions designed to gather information 
about the current professional career status of each 
respondent. If respondents indicate that they are in an 
active duty status, they are asked to give information such 
as warfare community, rank, technical nature of current job 
and career intentions. If respondents indicate that they are 
working in the civilian professional workforce, they are 
asked to provide information such as current title and/or 
position, name of employer, technical nature of current job 
and employing organization, and number of years served in 
current position. 

A field of 24 questions was designed for the second 
section of this survey, with the intention of addressing 
ABET's accreditation criteria. Specifically, these questions 
were developed in order to ascertain: (a) The level of 
preparedness that each graduate believes was achieved during 
their undergraduate engineering education with respect to 
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ABET's Criterion 3 'Program Outcomes' ('a' through 'k') ; 
and (b) The degree to which each of the 'a' through 'k' 
criteria is relevant to their current positions. 

According to Criterion 3, ABET requires that an 
institution be able to produce, assess and document the 
achievement of designated "skills, knowledge and behaviors" 
(ABET, 2005, p. 1) that a student should achieve by the time 
of graduation. These skills, knowledge and behaviors are 
further categorized as items "a" through "k": (a) an ability 
to apply knowledge of mathematics, science and engineering; 
(b) an ability to design and conduct experiments, as well as 
to analyze and interpret data; (c) an ability to design a 
system, component, or process to meet desired needs within 
realistic constraints such as economic, environmental, 
social, political, ethical, health and safety, 
manufacturability, and sustainability; (d) an ability to 
function on multi-disciplinary teams; (e) an ability to 
identify, formulate and solve engineering problems; (f) an 
understanding of professional and ethical responsibility; 
(g) an ability to communicate effectively; (h) the broad 
education necessary to understand the impact of engineering 
solutions in a global, economic, environmental and societal 
context; (i) recognition of the need for, and ability to 
engage in life-long learning; (j) a knowledge of 
contemporary issues; and (k) an ability to use the 
techniques, skills, and modern engineering tools necessary 
for engineering practice. 

The second section of the survey includes questions 
that identify the above skills or capabilities and asks 
respondents to indicate both the level of applicability of 
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those skills to their professional jobs (12 questions) and 
the level of preparation in each of those skill areas 
afforded them by their undergraduate engineering education 
(12 questions). Respondents were asked to indicate the level 
of applicability and preparation on a five-point scale, 
ranging from "Not at all" to "Extremely" applicable or 
prepared. The table below illustrates the manner in which 
these 24 questions can be mapped to ABET's Criterion 3 
"Program Outcomes." 

Table 2: Mapping Survey Questions to ABET Criterion 3 

"Program Outcomes" 


Survey Question 

Addresses which ABET Criterion 3 
"Program Outcome" 

Ability to apply knowledge of math, 
engineering and science? 

a. an ability to apply knowledge 
of mathematics, science and 
engineering 

Ability to analyze and interpret 
data? 

b. ability to design and conduct 
experiments, analyze and 
interpret data 

Ability to identify, formulate and 
solve engineering problems? 

e. ability to identify, 
formulate and solve engineering 
problems 

Ability to design systems, 
components or processes to meet 
needs ? 

c. ability to design a system, 
component, or process to meet 
desired needs within realistic 
constraints 

Ability to design and conduct 
experiments ? 

b. ability to design and conduct 
experiments, analyze and 
interpret data 

Ability to use techniques, skills 
and tools in engineering practice? 

k. ability to use the 
techniques, skills and modern 
engineering tools necessary for 
engineering practice 

Expectation to be current in 
technologies related to your current 
field? 

j. a knowledge of contemporary 
issues 

Ability to prepare reports and 
documents ? 

g. ability to communicate 
effectively 

Ability to prepare and deliver 
professional presentations? 

g. ability to communicate 
effectively 

Ability to function on multi¬ 
disciplinary teams? 

d. an ability to function on 
multi-disciplinary teams 

Understanding of ethical and 
professional responsibility? 

f. an understanding of 
professional and ethical 
responsibility 
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Survey Question 

Addresses which ABET Criterion 3 
"Program Outcome" 


h. broad education needed to 
understand impact of engineering 
solutions in global / societal / 
economic context 

Recognition of the need to engage in 
life-long learning? 

h. broad education needed to 
understand impact of engineering 
solutions in global / societal / 
economic context 

i. recognition of need to engage 
in life - long learning 


For each of the 24 questions in the second section of 


the survey, a five-point Likert scale us utilized. The 
Likert scale values and corresponding meanings are clarified 
below in Table 3: 


Table 3: Likert Scale Values and Corresponding Meanings 


Niomber in Likert 

Scale 

Corresponding 
Meaning for 
'Preparedness' 
questions 

Corresponding 
Meaning for 
'Applicability' 
questions 

1 

Not prepared at All 

Not Important at all 

2 

Somewhat Prepared 

Somewhat Important 

3 

Adequately Prepared 

Fairly Important 

4 

Very well Prepared 

Very Important 

5 

Extremely well 
Prepared 

Extremely Important 


In addition to answering specific questions rating the 
levels of preparedness and applicability of ABET's program 
outcomes, respondents were also asked several other 
questions in the second section of the survey that may 
provide useful feedback when evaluating their undergraduate 
education. Specifically, respondents were asked to indicate 

1. How well their education, in general, prepared 
them for their career. 
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2. How many times per year they attend professional 
conferences. 

3. Whether they have attained, or intend to attain 
advanced engineering licensing (PE licensing). 

4. How well prepared they felt for graduate 
education. 

5. The highest level of education that they had 
attained. 

The third and final section of the survey consists of 
three questions that allowed respondents to provide 
qualitative feedback about their engineering majors program. 
Each respondent was asked to provide comments to identify 
and discuss the most useful and the least useful aspects of 
their engineering education at the Naval Academy. They were 
also asked to provide comments or suggestions for 
improvement of the engineering program at the Naval Academy. 

D. PROCEDURE 

The design of the survey instrument used for this study 
was the result of a collaborative effort between the 
Institutional Research, Planning and Assessment (IR) 
Department and the Engineering Department faculty at the 
U.S. Naval Academy. The IR Department, founded in 1992, was 
designed "for the purpose of evaluating and disseminating 
institutional data to stimulate positive changes in the 
admissions and education processes at USNA. IR is the 
single source of evaluated information on Midshipman and 
graduate performance" (USNA, 2007). 
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As benchmarks for the development of this survey, IR 
and Engineering Department personnel used survey instruments 
from several other colleges and universities that had 
recently completed ABET accreditation visits. These 
institutions, including York College, Johns Hopkins 
University and Rutgers University, had utilized survey 
instruments in order to address ABET's new accreditation 
standards, namely Criterion 3 "Student Learning Outcomes." 

The survey developed by USNA (titled the "Graduate 
Performance Assessment Survey") was web-based. The IR 
Department obtained e-mail addresses from the Naval Academy 
Alumni Association for any graduates between the years of 
1985-2005 who had majored in engineering. Of the 7572 
graduates who fit this description, the Alumni Association 
had contact information for 4189 of them, as maintaining 
contact with the Alumni Association is voluntary for all 
graduates. (This translates into contact information for 
55.3% of the stated population.) The e-mail sent to each 
graduate contained the Website URL where the survey was 
located; respondents simply needed to 'click' on the URL, 
open the survey and begin entering data. 

The IR Department was responsible for survey 
administration and data integrity. Eollowing a brief pilot 
period, which was conducted in order to determine if the 
survey was complete, accessible and user-friendly, the 
survey was placed online for a period of one month between 
Eebruary and March of 2006. After the survey was taken 
offline, the IR Department pulled data from the electronic 
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responses and dropped them into a spreadsheet, and cleaned 
the data. The complete survey instrument is located in 
Appendix A. 

E. SAMPLE 

The sample for this study is a total of 1068 Naval 
Academy graduates who majored in engineering between 1985 - 
2005. Of the 4189 graduates from this time period for whom 
contact information was available and to whom the survey was 
sent, the 1068 respondents yield a response rate of 25.5%. 


Within the entire population of 7572 engineering majors from 
this time period, the 1068 respondents represent 14.1% of 
the population. 

All of the participants in this study were required to 
indicate their specific field of engineering study. The 
frequencies for each of the eight majors within USNA's 
Engineering Department are listed below in Table 4. Of these 
majors, all are still available for study at the Naval 
Academy with the exception of Marine Engineering, which was 
last offered in 1999. 


Table 4: Major 


Engineering 

Majors 

Number in 
Population 

Percent of 
Population 

Number in 
Sample 

Percent of 
Sample 

Aerospace 

(EAS) 

838 

11.10% 

106 

9.90% 

Aeronautics 

(EASA) 

505 

6.70% 

82 

7.70% 

Electrical 

(EEE) 

590 

7.80% 

92 

8.60% 
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Engineering 

Majors 

Number in 
Population 

Percent of 
Population 

Number in 
Sample 

Percent of 
Sample 

General (EGE) 

1020 

13.50% 

100 

9.40% 

Mechanical 

(EME) 

1376 

18.20% 

210 

19.60% 

Naval 

Architecture 

(ENA) 

431 

5.70% 

63 

5.90% 

Ocean 

Engineering 

(EOE) 

873 

11.50% 

111 

10.40% 

Systems (ESE) 

1684 

22.20% 

246 

23.00% 

Marine 

Engineering 

(ESP) 

255 

3.40% 

58 

5.40% 

Total 

7572 

100% 

1068 

100% 


When comparing the population to the sample, all 
engineering majors, with the exception of aerospace, ocean 
and general engineering, are slightly over-represented in 
the sample. The most over-represented major is marine 
engineering, where 2.0% more of the sample is comprised of 
marine engineers than is the case within the population. 
Conversely, the most under-represented major is general 
engineering; 13.5% of the population majored in general 
engineering, compared with only 9.4% of the sample (a 
difference of 4.1%) . With the exception of general 
engineering, the samples of all engineering majors represent 
the population within a tolerance of 2.0% or better. 

For purposes of this study, there is a need to track as 
much specific biographical information about each of our 
respondents as possible in order to maximize the 
opportunities for meaningful comparisons between distinct 
groups. For example, one focus of this study is on whether 
graduates are currently in an active duty military status or 
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a civilian status, and also which particular field of 
engineering each graduate majored in while attending the 
Naval Academy. For a list of demographic variables and 
descriptive statistics, see Appendix B. 

Tracking and identification information is made 
possible by having respondents indicate their alpha code. 
The alpha code is a unique identification number issued to 
Midshipmen on the day they report to the Naval Academy, 
which they maintain until graduation. Alpha codes are not 
duplicated or reissued regardless of graduation year. 


Table 5: Graduation Year 


Grad Year 

Sample 

Percent of Sample 

1985 - 1990 

321 

30 . 10 % 

1991 - 1995 

272 

25 . 50 % 

1996 - 2000 

270 

25 . 30 % 

2001 - 2005 

205 

19 . 20 % 

Total 

1068 

100 % 


The graduating classes of 1985-1990 are slightly over¬ 
represented in our sample, whereas the graduating classes of 
2001-2005 are slightly under-represented. There is a good 
sample distribution across the graduating classes of 1991- 
2000, which represents approximately 50% of the sample. 


Table 6: Civilian / Military Status 


Status 

Frequency 

Percent of 
Sample 

Civilian 

546 

51.1 

Active Duty Military 

522 

48.9 

Total 

1068 

100.0 
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This is close to an ideal sample distribution between 
civilian and military respondents, as a good bit of analysis 
is conducted with the intent of comparing civilian to active 
duty military respondents. The sample distribution is 
roughly equal, with civilian and active duty military both 
representing about half of the sample. 

With few exceptions, all graduates of the Naval Academy 
are required to accept commissions as either Navy or Marine 
Corps officers following graduation. Therefore, all 
respondents served at least one year on active duty. As a 
general rule, graduates will owe a minimum of 5 years on 
active duty, although this number frequently changes based 
on many factors including warfare community, needs of the 
service, and health status of the service member. All 
respondents indicated the total amount of time they spent on 
active duty, which ranged in time from 1 year to 8 years. In 
the sample, 90.1% of the respondents served between 3 and 5 
years on active duty. 

Respondents are also asked to indicate the warfare 
community in which they served (or serve). The vast majority 
of the sample (81.5%) indicated that they served in one of 
the four largest unrestricted line warfare communities that 
are available to Naval Academy graduates, namely naval 
aviation, surface warfare, submarine warfare and the Marine 
Corps. Approximately 9% of respondents serve (or served) in 
the Civil Engineering Corps or as Engineering Duty Officers. 

Survey respondents were asked to answer questions in 
order to provide details as to the nature of their current 
job. Eor example, out of 515 active duty military 
respondents, 69.3% indicated that the nature of their 
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current job is technical, with the remaining 30.7% 
indicating that their current job is non-technical. 

In contrast, 57.0% of civilian respondents indicated 
that their current occupation is technical in nature, and 
43.0% indicated that their current occupations are non¬ 
technical in nature. (To see further information concerning 
this breakdown between technical and non-technical 
positions, see Appendix B.) 

Civilian respondents were asked to indicate the level 
of their current position within their current organization, 
ranging from entry level to senior executive (i.e.. 
President, CEO). Again, a complete description of this 
breakdown can be found in Appendix B. Of the civilian 
respondents, 7.4% indicated that they are the Senior 
Executive of their organization; 40.5% indicated that they 
work as upper management, and 43.8% indicated that they work 
as mid-level management. A small percentage (3.3%) indicated 
that they are currently working at entry level positions. 

F. ANALYSIS 

The majority of the analysis for this study focuses on 
the entire population, with an effort being made to 
determine the level of applicability and preparedness that 
respondents assign to each of the engineering related skills 
and abilities. Additional analysis focuses on comparing 
military and civilian respondents, and also on comparing 
respondents from differing engineering majors. The portion 
of the survey instrument that is utilized for comparison 
analyses are the questions, described earlier in this 
chapter, modeled after ABET's "Criterion 3" (a - k) . These 
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questions focus on specific student learning outcomes, and 
allow respondents to comment on levels of applicability of 
certain skills to their profession and adequacy of 
preparation attained during undergraduate study. 

Independent samples T-tests are conducted in order to 
compare military and civilian respondents. The goal is to 
identify statistically significant differences between 
military and civilian respondents with respect to how they 
view the applicability of certain engineering related skills 
to their jobs, and also to identify significant differences 
in levels of preparation. All significance criteria (for all 
analyses) will be p < .05. 

For both civilian and military respondents, paired 
sample T-tests are conducted in order to determine if there 
are statistically significant differences between the levels 
of preparation and the levels of applicability for each of 
the stated engineering skills. In other words, the paired 
sample test is used to determine whether respondents within 
each of the two groups (military or civilian) believed that 
their level of preparedness for each engineering skill was 
high or low relative to the perceived level of skill 
applicability. 

Analysis of variance (ANOVA) tests are conducted in 
order to compare respondents across the nine different 
engineering majors. Specifically, the ANOVA test is utilized 
to determine if there is a statistically significant 
difference between respondents of different engineering 
majors with regard to how well they feel they were prepared 
in engineering related skills, or how applicable they 
believe those skills to be. 
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IV. DATA ANALYSIS 


A. ENTIRE SAMPLE: APPLICABILITY VS. PREPAREDNESS 

All respondents were asked to indicate the extent to 
which they felt that each of 12 different engineering 
related skills were applicable to their current profession. 
They were also asked to indicate the extent to which they 
felt that their undergraduate engineering education had 
prepared them to perform these engineering related skills. 
Answers to these questions ranged from 1 ( 'Not important at 

all' or ' Not prepared at all') to 5 ( 'Extremely important' 

or 'Extremely well prepared' ) . The results for the entire 
sample are summarized in Table 7. 

Table 7: Sample Mean Values for Applicability and 

Preparedness 


Engineering Skill or Ability 

Applicability 

Preparedness 

Mean 

N 

Mean 

N 

Apply knowledge of math, engineering and 
science. 

3.84 

1037 

4.39 

1065 

Ability to analyze and interpret data. 

4.35 

1015 

4.48 

10 66 

Identify, formulate, and solve engineering 
problems. 

3.09 

978 

4.27 

1065 

Design systems, components, or processes 
to meet needs. 

2.97 

980 

3.79 

1063 

Design and conduct experiments. 

2.34 

934 

3.78 

1062 

Use techniques, skills, and tools in 
engineering practice. 

3.03 

962 

4.07 

1060 

Expectation to be current in technologies 
related to your career field. 

3.89 

1001 

3.74 

1060 

Prepare reports and documents. 

4.38 

991 

4.26 

1063 

Prepare and deliver professional 
presentations . 

4.48 

992 

4.11 

1062 

Function on multi-disciplinary teams. 

4.4 

982 

4.35 

1062 

Understanding of ethical and professional 
responsibility. 

4.74 

964 

4.67 

1064 

Recognition of the need to engage in life¬ 
long learning. 

4.44 

988 

4.38 

1060 
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The six skills reported to be most applicable (means > 
4.3) to the respondents' current jobs were: "ability to 
analyze and interpret data;" "prepare reports and 
documents;" "prepare and deliver professional 
presentations;" "function on multi-disciplinary teams;" 
"understanding of ethical and professional responsibility;" 
and "recognition of the need to engage in life-long 
learning." Four skills had mean ratings at or below the 
mid-point rating of 3 for applicability: "Identify, 
formulate, and solve engineering problems;" "design systems, 
components, or processes to meet needs;" "design and conduct 
experiments;" and "use techniques, skills, and tools in 
engineering practice." 

All but three skill areas received ratings >4.0 in 
terms of how well the degree program prepared them. The 
highest rated skill area was "understanding of ethical and 
professional responsibility" (mean = 4.67). The three skill 
areas with the lowest ratings of preparation were: "design 
systems, components, or processes to meet needs"; "design 
and conduct experiments"; and "expectation to be current in 
technologies related to your current field." It is 
noteworthy that the first two of these lowest-rated skills 
also receive the lowest ratings of applicability to their 
current jobs. 

For 6 of the 12 skills, respondents indicated, on 
average, that their level of preparation met or exceeded the 
level of ability required in that skill area in their 
current job or profession. Notably, respondents felt well- 
prepared to identify, formulate and solve engineering 
problems; design systems, components, or processes to meet 


40 



needs; design and conduct experiments; and use techniques, 
skills, and tools in engineering practice, yet they did not 
view these skills as highly relevant to their current 
positions. 

For the other six skills, respondents indicated, on 
average, that their level of preparation did not meet the 
level of ability required in that skill area. Notably, 
respondents felt under-prepared with regard to their ability 
to prepare reports and documents, and to prepare and deliver 
professional presentations. 

B. EFFECTS OF APPLICABILITY ON PREPAREDNESS 

The intention of this section is to examine the results 
only for the survey respondents who believe that these 12 
engineering related skills are applicable to their current 
job or profession. For those respondents who reported that 
the skills are applicable, levels of perceived preparedness 
were analyzed. Those respondents who indicated that a given 
skill was "not important at all" or "somewhat important" 
were eliminated from the analysis of each skill category. 
Only those respondents who indicated that a given skill was 
"fairly important," "very important" or "extremely 
important" were included in this analysis. The results are 
summarized in Table 8. 
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Table 8: How Well USNA Degree Prepared Graduates in Skill 
Areas Applicable to Current Work 


Engineering Skill or Ability 

PREPAREDNESS 

MEAN 

N 

Apply knowledge of math, engineering and science. 

4 . 42 

893 

Ability to analyze and interpret data. 

4.48 

944 

Identify, formulate, and solve engineering problems. 

4.35 

616 

Design systems, components, or processes to meet needs. 

3.94 

563 

Design and conduct experiments. 

3.92 

366 

Use techniques, skills, and tools in engineering 
practice. 

4.19 

567 

Expectation to be current in technologies related to 
your career field. 

3.77 

841 

Prepare reports and documents. 

4.28 

926 

Prepare and deliver professional presentations. 

4.1 

939 

Function on multi-disciplinary teams. 

4.36 

914 

Understanding of ethical and professional 
responsibility. 

4.67 

943 

Recognition of the need to engage in life-long learning. 

4.38 

94 6 


All skill categories have preparedness ratings greater 
than 3.75, and all but three are greater than 4.0. For 6 of 
the 12 skill categories, more than 900 respondents indicated 
high levels of applicability: 'ability to analyze and 
interpret data;" "prepare reports and documents;" "prepare 
and deliver professional presentations;" function on multi¬ 
disciplinary teams;" "understanding of ethical and 
professional responsibility;" and "recognition of the need 
to engage in life-long learning." Of these six skills, the 
abilities to "prepare reports and documents" and "prepare 
and deliver professional presentations" received the lowest 
ratings for preparedness. 

The 'N' values themselves are telling in this analysis. 
Approximately 20% to 50% fewer respondents indicated high 
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levels of applicability for four of the 12 skill areas (as 
compared with the other skill categories) : "identify, 
formulate, and solve engineering problems;" "design systems, 
components, or processes to meet needs;" "design and conduct 
experiments;" and "use techniques, skills, and tools in 
engineering practice." But for those who did feel that these 
skills were applicable, their perceived levels of 
preparedness are higher than the overall sample means 
reported in the previous table. 

It is also interesting to note that, in general, as the 
number of respondents who believed that a given skill was 
applicable to their current profession increases, the 
average indication of preparedness for that skill also 
increases. The largest number of respondents indicated that 
the skills "understanding of ethical and professional 
responsibility;" "ability to analyze and interpret data" and 
"recognition of the need to engage in life-long learning" 
were applicable to their current jobs, and these skills were 
also rated highest in terms of preparedness. 

C. PAIRED DIFFERENCES FOR ENGINEERING SKILLS 

In this section, each of the 12 "preparedness" 
questions was paired with their corresponding 
"applicability" questions. A paired samples t-test was 
conducted in order to determine any significant difference 
between the level of preparedness and the level of 
applicability for each of the engineering related skills. 
The results of the paired samples t-test (using the entire 
sample) are shown in Table 9. 
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Table 9: 


Mean Differences Between 'Preparedness' and 
'Applicability' 


Engineering Skill or Ability 

N 

Mean Diff ('Preparedness' 

- 'Applicability' ) 

Sig. 

Apply knowledge of math, 
engineering and science. 

1062 

0.49 

0.000 

Ability to analyze and interpret 
data. 

1060 

0.06 

0.099 

Identify, formulate, and solve 
engineering problems. 

1057 

0 . 96 

0.000 

Design systems, components, or 
processes to meet needs. 

1058 

0 . 60 

0.000 

Design and conduct experiments. 

1057 

1.00 

0.000 

Use techniques, skills, and tools 
in engineering practice. 

1054 

0.77 

0.000 

Expectation to be current in 
technologies related to your 
career field. 

1052 

-0.26 

0.000 

Prepare reports and documents. 

1057 

-0.22 

0.000 

Prepare and deliver professional 
presentations . 

1058 

-0.47 

0.000 

Function on multi-disciplinary 
teams. 

1056 

-0.17 

0.000 

Understanding of ethical and 
professional responsibility. 

1056 

-0.18 

0.000 

Recognition of the need to engage 
in life-long learning. 

1054 

-0.17 

0.000 


A positive value for the mean difference indicates that 
respondents, on average, gave higher ratings to their 
perceived level of preparedness relative to the rating of 
applicability. Conversely, a negative value for the mean 
difference indicates that respondents, on average, gave 
lower ratings to their perceived level of preparedness 
relative to the rating of applicability for that skill. 


For 

the 

skills listed below. 

the mean 

difference 

between 

the 

level of preparedness 

and 

the 

level 

of 

applicabi 

lity 

was positive and statistically 

significant 

(P 


< .05) . That means that respondents (on average) gave high 
ratings for preparedness in these skill areas relative to 
the level of skill applicability: 
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1. Ability to apply knowledge of math, engineering and 
science. 

2. Ability to identify, formulate and solve engineering 
problems. 

3. Ability to design systems, components or processes 
to meet needs. 

4. Ability to design and conduct experiments. 

5. Ability to use techniques, skills and tools in 
engineering practice. 

For the skills listed below, the mean difference 
between the level of preparedness and the level of 
applicability was negative and statistically significant (p 
< .05), meaning that respondents (on average) indicated low 
ratings for preparedness in these skill areas relative to 
the level of skill applicability: 

1. Expectation to be current in technologies related to 
your current field. 

2. Ability to prepare reports and documents. 

3. Ability to prepare and deliver professional 
presentations. 

4. Ability to function on multi-disciplinary teams. 

5. Understanding of ethical and professional 
responsibility. 

6. Recognition of the need to engage in life-long 
learning. 
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D. EFFECTS OF JOB TYPE ON PERCEIVED PREPAREDNESS 

All respondents were asked the general question, "How 
well did your engineering major at USNA prepare you for your 
current career?" Responses ranged from 1 ( 'Not Prepared at 

All') to 5 ('Extremely Well Prepared'). For this section, an 

independent samples T-test was conducted in order to 
determine whether respondents who reported that their 
current job is technical in nature believed that they are 
better prepared than those who reported that their current 
job is non-technical in nature. The grouping variable is a 
binomial variable that indicates whether a respondent's job 
is technical or non-technical, and the test variable is the 
respondents' rating of their level of preparedness. The 
results are summarized in Table 10. 

Table 10: How Well did Major Prepare for Current Career? 

(Technical vs. Non-Technical) 


Job Description 

N 

Mean 

Std. Dev 

Technical 

664 

4.05 

0.867 

Non-Technical 

388 

3.79 

1.019 


The overall sample mean for this 'preparedness' 
variable is 3.95. The results of the t-test indicate that 
there is, in fact, a statistically significant difference 
(F=25.51, p < .05) between respondents in technical jobs and 
those in non-technical jobs with regard to how they perceive 
their level of preparation. In particular, respondents who 
are in technical jobs give higher ratings to their level of 
preparation than those who are in non-technical jobs. 
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E. EFFECTS OF MAJOR ON PERCEIVED PREPAREDNESS 

Respondents were asked the general question, "How well 
did your engineering major at USNA prepare you for your 
current career?" Responses ranged from 1 ("Not Prepared at 
All") to 5 ("Extremely Well Prepared"). A one-way analysis 
of variance (ANOVA) test was conducted, using the major as 
the grouping variable and level of preparedness as the test 
variable. This test was conducted to determine if there was 
a significant difference in the means between each group. 
The results are summarized in Table 11. 


Table 11: How Well did Major Prepare for Current Career? 


Major 

N 

Mean Value 

Std. Dev. 

Systems Engineering 

245 

4.00 

0.896 

Mechanical Engineering 

210 

3.90 

0.890 

Ocean Engineering 

111 

3.96 

0.972 

Aerospace Engineering 

104 

3.91 

0.946 

General Engineering 

100 

3.75 

0.925 

Electrical Engineering 

92 

4.09 

0.898 

Aeronautics Engineering 

82 

4.15 

0.904 

Naval Architecture 

62 

3.95 

1.093 

Marine Engineering 

58 

3.86 

0.999 

Total 

1064 

3.95 

0.932 


Homogeneity of variance between the major groups was 
verified to be true, and the results of the ANOVA test 
indicate that there is no statistically significant 
difference (F=1.55, p < .05) between the major groups with 

respect to how prepared, in general, each group felt. 

F. PREPARATION FOR ADVANCED EDUCATION 

Respondents were asked to indicate whether or not they 
had attained any forms of advanced education during the time 


47 




since their graduation from the Naval Academy, ranging from 
partial completion of a graduate degree to the completion of 
a Doctorate degree. They were also asked to indicate the 
extent to which they felt that their undergraduate education 
had prepared them for graduate education. (This question was 
posed to all respondents, whether they completed further 
education or not.) To indicate their level of preparation, 
respondents chose a number between 1 ('not at all prepared') 
and 5 ( 'extremely well prepared') . The descriptive 
statistics for this analysis are shown in Table 12 below. 
The first column indicates the highest level of education 
attained, and the mean value indicates the extent to which 
that group felt prepared for graduate education. 


Table 12: Feeling of Preparedness for Advanced Education 


EDUCATION LEVEL 

N 

MEAN 

STD. DEV. 

PhD or equivalent 

29 

4.03 

0.906 

Master's Degree or equivalent 

515 

4.06 

0.914 

Some graduate work 

114 

3.98 

0.902 

Bachelor's Degree 

393 

3.74 

0.803 


A one-way analysis of variance (ANOVA) test was 
conducted in order to determine whether a significant 
difference existed between group means. The grouping 
variable was the highest level of education attained, and 
the test variable was the level of preparedness for graduate 
education. The results of this test indicate that there are 
significant differences (F=9.89, p < .05) between at least 
two of these group means. 

In order to determine which groups differ, a post hoc 
LSD test was conducted. The results indicate that those 
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respondents with a master's degree felt significantly more 
prepared for graduate education than those respondents who 
have not furthered their education beyond a bachelor's 
degree (p < .05) . Additionally, those respondents who have 
completed some graduate work felt significantly more 
prepared for graduate education than those respondents who 
have not furthered their education beyond their Bachelor's 
degree (p< .05). These results indicate that those who 

pursue graduate education have a higher perceived 
preparation than those who have only completed the 
bachelor's degree. 

G. RELATIONSHIP BETWEEN CAREER AND EDUCATION 

Civilian respondents were asked to indicate the extent 
to which their current job is related to their undergraduate 
engineering education at the Naval Academy. Possible answers 
were 'Not At All,' 'Somewhat' or 'To a Great Extent.' For 
the purposes of this specific analysis, these responses were 
recoded into a new binomial numeric variable, in which those 
who answered 'Not At All' are classified as having no 
relationship between undergraduate education and job, and 
those who answered 'Somewhat' or 'To a Great Extent' are 
classified as having a relationship between undergraduate 
education and job. 

All respondents were asked the general question, "How 

well did your engineering major at USNA prepare you for your 

current career?" Responses ranged from 1 ("Not Prepared at 

All") to 5 ("Extremely Well Prepared"). An independent 

samples t-test was conducted in order to determine whether 

those civilian respondents who felt that their education 

applied to their current job felt more or less prepared than 
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those respondents who did not see any relationship between 
their education and their job. The descriptive statistics 
for this analysis are displayed in Table 13. 


Table 13: "How Well did Major Prepare for Current Career" 

(Civilians Only) 


Job Related to Education? 

N 

Mean 

Related 

346 

4.06 

Not Related 

195 

3.62 


The results of the t-test indicate that there is a 
significant difference (t=-4.92, p < .05) between these two 
groups. Here, those civilian respondents who believed that 
their job was related to their engineering education felt 
significantly better prepared for their career than those 
respondents who report no relationship between job and 
education. 

H. CIVILIAN/MILITARY: INDEPENDENT SAMPLE T-TEST RESULTS 

1. Overall Feeling of Preparedness 

Respondents were asked the general question, "How well 
did your engineering major at USNA prepare you for your 
current career?" Responses ranged from 1 ("Not Prepared at 
All") to 5 ("Extremely Well Prepared"). An independent 
samples t-test was conducted, using a binomial grouping 
variable that indicates whether respondents are civilian or 
military, and overall level of preparedness as the test 
variable. This test was conducted to determine if there was 
a significant difference in the means for the civilian and 
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military groups, with respect to their level of perceived 
overall preparedness. The results are presented in Table 14. 


Table 14: How Well did Major Prepare for Current Career? 

(Civilian / Military) 


Status 

N 

Mean 

Std. Dev. 

Civilian 

543 

3.90 

0.963 

Active Duty Military 

521 

4.01 

0.895 


The significance level of the t-test was 0.065. Thus, 
it cannot be concluded that there is a significant 
difference (p < .05) between the two group means (civilian 
and military), although there is a significant difference at 
the P < .10 level, with military rating themselves as better 
prepared than their civilian employed counterparts. 

2. Preparedness in Applicable Engineering Related 
Skills 

An independent samples t-test was conducted, using a 
binomial grouping variable that indicates whether 
respondents are civilian or active duty military. The test 
variables were the 12 "preparedness" questions that allowed 
respondents to indicate their level of preparedness with 
respect to twelve different engineering related skills. 

The intention of this analysis is to examine the 
responses provided by survey respondents who believe that 
these engineering related skills are applicable to their 
current job or profession. With this in mind, those 
respondents who indicated that a given skill was "not 
important at all" or "somewhat important" were eliminated 
from the analysis of each skill category prior to each 
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respondents who 


independent sample test. Only those 
indicated that a given skill was "fairly important," "very 
important" or "extremely important" were included in this 
analysis. 

The results of this analysis are summarized in Table 
15. For 9 of the 12 engineering related skills, there was no 
statistically significant difference (p < .05) between 

military and civilian with respect to preparedness. For the 
ability to "apply knowledge of math and science;" "function 
on multi-disciplinary teams;" and "understand ethical and 
professional responsibility," civilians indicated a higher 
level of preparedness (p < .05). 

Table 15: How well USNA Degree Prepared Graduates in Skill 
Areas Applicable to Current Profession 


Engineering Skill or 
Ability 

Civilian 

Military 

t-test sig. 


Mean 

N 

Mean 

N 


Apply knowledge of math, 
engineering and science. 

4.48 

469 

4.35 

424 

. 002 

Ability to analyze and 
interpret data. 

4.52 

494 

4.44 

450 

.072 

Identify, formulate, and 
solve engineering 
problems. 

4.37 

330 

4.32 

286 

.390 

Design systems, 
components, or processes 
to meet needs . 

3.89 

334 

4 . 02 

229 

.065 

Design and conduct 
experiments. 

3.86 

220 

4.01 

146 

. 122 

Use techniques, skills, 
and tools in engineering 
practice. 

4.22 

300 

4.14 

267 

.213 

Expectation to be current 
in technologies related to 
your career field. 

3.75 

433 

3.80 

408 

.458 

Prepare reports and 
documents. 

4.29 

470 

4.26 

456 

.470 

Prepare and deliver 
professional 
presentations. 

4.07 

473 

4 . 13 

466 

.317 

Function on multi¬ 
disciplinary teams. 

4.44 

473 

4.28 

441 

. 002 

Understanding of ethical 
and professional 
responsibility. 

4.74 

477 

4 . 60 

466 

. 000 

Recognition of the need to 
engage in life-long 
learning. 

4.43 

480 

4.34 

466 

.090 
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3 . 


Technical vs. Non-Technical: Do Civilians Differ 
from Military? 


In an earlier section, an independent samples t-test 
was conducted in order to determine whether respondents who 
worked in technical jobs felt differently about their level 
of preparation than those who worked in non-technical jobs. 
The results of that analysis indicated that, in general, 
those in technical jobs felt significantly better prepared 
(p < .05) than those in non-technical jobs. 

In this section, the intention is to further this 
analysis by comparing civilian and military respondents. All 
respondents were asked the general question, "How well did 
your engineering major at USNA prepare you for your current 
career?" Responses ranged from 1 ("Not Prepared at All") to 
5 ("Extremely Well Prepared") . 

In the analysis, respondents are first grouped 
according to whether they are military or civilian. Two 
independent samples t-tests were conducted (one for the 
civilian group and one for the military group) in order to 
determine whether respondents who felt that their current 
jobs were technical in nature believed that they were better 
prepared than those who felt that their current job is non- 


technical 

in 

nature. The 

grouping variable is a 

binomial 

variable 

that 

indicates 

whether a respondent's 

job 

is 

technical 

or 

non-technical 

, and the test variable 

is 

the 


respondents' rating of their level of preparedness. The 
results are summarized in Tables 16 and 17. 
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Table 16: How well did Major Prepare for Current Career? 
(Technical vs. Non-technical for Civilian Respondents) 


Civilian Job Description 

N 

Mean 

Std. Dev. 

Technical 

308 

3.97 

0.898 

Non-Technical 

230 

3.81 

1.048 


Table 17: How well did Major Prepare for Current Career? 
(Technical vs. Non-technical for Military Respondents) 


Military Job Description 

N 

Mean 

Std. Dev. 

Technical 

356 

4.11 

0.836 

Non-Technical 

158 

3.77 

0 . 977 


For the civilian respondents, the significance of the 
t-test was 0.055. Thus, it cannot be concluded that a 
significant difference (p < .05) exists between technical 
and non-technical jobs within the civilian group. Within the 
active duty military group, however, the significance value 
was smaller than .0001, indicating that a significant 
difference does exist between technical and non-technical 
jobs within the military. In particular, those military 
respondents who felt that their jobs were technical in 
nature believed, on average, that they were better prepared 
for their job due to their undergraduate engineering 
education. In both cases (civilian and military), the 
average value for "level of preparedness" was higher for 
those who believed their jobs to be technical, although the 
difference is statistically significant only for the 
military group. 


54 




I. CIVILIAN/MILITARY: PAIRED SAMPLES T-TEST RESULTS 

For the paired samples t-tests, the cases were split 
into two categories (civilian and military). Each of the 12 
"preparedness" questions were paired with their 
corresponding "applicability" questions, and the paired 
samples t-test was conducted in order to determine any 
significant difference between the level of preparedness and 
the level of applicability for each of the engineering 
related skills. For these tests, the N value for the 
civilian group ranged from 534 to 541, and the N value for 
the military group ranged from 517 to 521. The results are 
summarized in Table 18. 

All but three differences are significant at the p< .05 
level. A positive value for the mean difference indicates 
that respondents, on average, gave higher ratings to their 
perceived level of preparedness relative to the rating of 
skill applicability. Conversely, a negative value for the 
mean difference indicates that respondents, on average, gave 
lower ratings to their perceived level of preparedness 
relative to the rating of applicability for that skill. 
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Table 18: Paired Sample T-Test (Mean Differences for 

Civilian/Military) 


Engineering Skill / Ability 

Mean Difference 
(Preparedness/Applicability) 

CIVILIAN 

MILITARY 

Ability to apply knowledge of 
math, engineering and science. 

0.47 

0.51 

Ability to analyze and 
interpret data. 

No significant 
difference 

0.2 

Ability to identify, formulate 
and solve engineering problems. 

0 . 94 

0 . 98 

Ability to design systems, 
components or processes to meet 
needs. 

0.27 

0 . 94 

Ability to design and conduct 
experiments. 

0.87 

1.14 

Ability to use techniques, 
skills and tools in engineering 
practice. 

0.73 

0.82 

Ability to use technologies 
related to your career field. 

-0.38 

-0.13 

Ability to prepare reports and 
documents. 

-0.38 

No significant 
difference 

Ability to prepare and deliver 
professional presentations. 

-0.58 

-0.35 

Ability to function on multi¬ 
disciplinary teams. 

-0.29 

No significant 
difference 

Ability to accept ethical and 
professional responsibility. 

-0.16 

-0.21 

Importance of engaging in life¬ 
long learning. 

-0.15 

-0.19 


For civilian respondents, there was no significant 
difference between their level of applicability and their 
level of preparedness for their ability to apply knowledge 
of math, engineering and science. 

For military respondents, there was no significant 
difference between their level of applicability and their 
level of preparedness for the following engineering skills: 
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1. Ability to prepare reports and documents. 

2. Ability to function on multi-disciplinary teams. 

For all other paired questions, there were significant 
differences between the mean values of applicability and 
preparedness, for both military and civilian respondents. 

For the skills listed below, the difference between the 
level of preparedness and the level of applicability was 
positive, meaning that respondents (on average) gave high 
ratings for preparedness in these skill areas relative to 
the level of skill applicability: 

1. Ability to apply knowledge of math, engineering and 
science. 

2. Ability to analyze and interpret data. 

3. Ability to identify, formulate and solve engineering 
problems. 

4. Ability to design systems, components or processes 
to meet needs. 

5. Ability to design and conduct experiments. 

6. Ability to use techniques, skills and tools in 
engineering practice. 

For the skills listed below, the difference between the 
level of preparedness and the level of applicability was 
negative, meaning that respondents, on average, gave low 
ratings for preparedness in these skill areas relative to 
the level of skill applicability: 

1. Expectation to be current in technologies related 
to your current field. 
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2 . 


Ability to prepare reports and documents. 


3. Ability to prepare and deliver professional 
presentations. 

4. Ability to function on multi-disciplinary teams. 

5. Understanding of ethical and professional 

responsibility. 

6. Recognition of the need to engage in life-long 
learning. 
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V. 


SUMMARY, CONCLUSION AND RECOMMENDATIONS 


A. SUMMARY 

There were several purposes to this research. The 
primary purpose was to determine the extent to which Naval 
Academy graduates who majored in engineering believe their 
undergraduate education to be relevant to their current 
profession, utilizing the Accreditation Board of Engineering 
and Technology (ABET) "student learning outcomes" (ABET, 
2005, p. 1) as the basis for assessing relevance. The 
intention was to assess both the applicability of these 
particular skills to graduates' current professions, as well 
as the perceived levels of preparedness in these skill areas 
afforded respondents due to their undergraduate education. 

A secondary purpose was to draw meaningful comparisons 
between different groups within the sample. These comparison 
groups include military and civilian, technical and non¬ 
technical jobs, and comparison across the different 
engineering majors. 

A third purpose of this research was to establish a 
body of baseline data about engineering majors who have 
graduated from the Naval Academy throughout the past 20 
years. The intention was to gather a broad base of data 
about what professions or jobs graduates are currently doing 
and how their undergraduate education has factored into 
their careers. 

An overarching goal of this research is to examine the 
relationship between engineering education at the Naval 

Academy and the impact that education has on graduates after 
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they graduate and begin their professional lives. 
Respondents have provided valuable information and insight 
based on their personal experience, about what skills have 
been applicable to their professional lives and how they 
perceive their level of preparation due to their engineering 
education. The emphasis is on identifying the strengths of 
the educational program, as well as identifying potential 
areas for future improvement. 

B. CONCLUSION 

Overall, Naval Academy graduates who majored in 
engineering are generally satisfied with the quality of 
their undergraduate education. They also feel that what they 
studied in their engineering program is applicable to what 
they do today. 

For 6 of the 12 engineering related skills or abilities 
that were analyzed in this research, respondents indicated 
very high levels of importance to their current work. The 
six most important skills were: "understanding of ethical 
and professional responsibility;" "prepare and deliver 
professional presentations;" "recognition of the need to 
engage in life-long learning;" "function on multi¬ 
disciplinary teams;" and "ability to analyze and interpret 
data." All of these skills received ratings of importance 
between 4.38 and 4.74 (i.e., between "very important" and 
"extremely important"). 

The skills that were rated as least important to 
respondents were: "design and conduct experiments;" "design 
systems, components or processes to meet needs;" "use 
techniques, skills and tools in engineering practice;" and 
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These 


"identify, formulate and solve engineering problems, 
skills received ratings of importance between 2.34 and 3.09 
(i.e., between "somewhat important" and "fairly important"). 
For all of these skills, respondents (on average) indicated 
high levels of preparedness relative to the level of 
applicability for the particular skill. 

In general, it seems that respondents gave lower 
ratings of importance to skills that could be seen as 
particular to technical work. Respondents gave higher 
ratings of importance to skills that are more universally 
applicable. 

Respondents felt well prepared for the abilities to 
"design and conduct experiments" and "identify, formulate 
and solve engineering problems," yet reported low ratings 
for the applicability of these skills to their current job. 

Conversely, respondents rated their preparedness low 
relative to the level of applicability in the skill areas of 
"expectation to be current in technologies related to 
current career field;" "ability to prepare and deliver 
professional presentations;" and ability to "prepare written 
reports and documents;" "ability to function on multi¬ 
disciplinary teams;" "understanding of ethical and 
professional responsibility" and "recognition of the need to 
engage in life-long learning." For these skills, respondents 
reported high levels of preparedness, but they reported 
extremely high levels of applicability, especially for the 
skills related to preparing and delivering presentations and 
preparing reports and documents. 

As could be expected, respondents who indicated that 
their current job or profession was technical in nature felt 
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that their undergraduate engineering education had prepared 
them for their job to a significantly higher degree than 
those respondents who indicated that their current job or 
profession was non-technical in nature. This result is not 
surprising, considering the technical nature of the 
respondents' undergraduate education. 

In general, which of the engineering fields a 
respondent majored in had no significant effect on how well 
prepared they felt for their current job due to their 
undergraduate education. Sample means ranged from 4.15 
(aeronautics) to 3.75 (general engineering), with an overall 
average of 3.95 (i.e., "very well prepared"); however, none 
of the differences between major groups were statistically 
significant. 

This congruence across majors may be a result of the 
fact that regardless of which particular field a respondent 
chose to major in, all were required to take a high 
percentage of core curriculum courses that are common to all 
engineering majors. While each major offers many technical 
courses that are specific to that particular field, there 
are a large number of common courses that provide all 
engineering majors with similar technical and engineering 
related skills. 

Respondents who had gone on to either complete a 
master's degree or begin working on a master's degree felt 
significantly better prepared for graduate education than 
those respondents who had not advanced their formal 
education beyond their bachelor's degree. This result seems 
to indicate that USNA graduates who majored in engineering 
underestimate the degree to which they are prepared to 
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pursue advanced education in the form of a master's program. 
However, it should be noted that respondents were not asked 
to provide any specific information about the nature of 
their master's program. 

1. Civilian / Military Comparison 

On average, respondents who were still active duty 
military gave slightly higher ratings to their feelings of 
preparedness for their current job than those respondents 
who have transitioned to the civilian work force. The 
difference between military and civilian was statistically 
significant at the p < .10 level, but not at the p < .05 
level. 

This difference in perceived level of preparedness 
could be driven in part by the fact that a good deal of the 
core curriculum offered at the Naval Academy is focused on 
military - specific education, such as courses in 
leadership, tactics, military history and navigation. These 
courses are designed specifically to prepare graduates for 
the time they will serve as officers in the Navy or Marine 
Corps. 

For 9 of the 12 specified engineering related skills, 
there was no significant difference between civilian and 
military respondents with respect to their perceived levels 
of preparedness. Interestingly, for the three skills where a 
significant difference did exist, civilians gave higher 
ratings to their level of preparedness than the military 
respondents. These skills were "the ability to apply 
knowledge of math, engineering and science;" "function on 
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multi-disciplinary teams;" and "understanding of 
professional and ethical responsibility." 

With respect to the perceived difference between level 
of preparedness and the level of applicability of each 
engineering related skill or ability, there are several 
notable differences between civilian and military 
respondents. Civilian respondents gave low ratings of 
preparedness to the skills of "preparing reports and 
documents" and "ability to function on multi-disciplinary 
teams" relative to the level of these skills' applicability. 
Military respondents showed no significant difference 
between preparedness and applicability with respect to these 
skills . 


It 

is 

interesting to 

note that 

although civilian 

respondents 

reported higher 

levels 

of 

preparedness 

than 

military 

respondents with 

respect 

to 

"the ability 

to 

function 

on 

multi-disciplinary teams. 

" civilians still 

felt 


under-prepared in this skill area, whereas military 
respondents did not. This can be attributed to civilian 
respondents that reported much higher levels of 
applicability (4.72) than military respondents (4.30) for 
this given skill. For all other skills, there were slight 
variations between civilian and military respondents, but in 
general, both groups reported similar results when comparing 
levels of applicability and preparedness. 

C. RECOMMENDATIONS AND LIMITATIONS 

One limitation to this study was caused by lack of 
comprehensive access to Naval Academy graduates who majored 
in an engineering field. Only those graduates who maintain 
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current contact information with the Naval Academy Alumni 
Association were able to be contacted, which certainly 
degrades the randomness of the sample. 

Another limitation is that this research was conducted 
in an entirely quantitative fashion. There is a large amount 
of qualitative data that was not analyzed. These data are 
from the part of the survey instrument that allowed 
respondents to write their own comments in response to 
several questions. Respondents were asked to comment on the 
most useful aspects of their engineering education, the 
least useful aspects of their engineering education, and 
finally to offer additional comments or suggestions for 
improvement of the program. The written answers to these 
questions will provide a good starting point for a 
qualitative assessment of the Naval Academy's engineering 
program, and would be an excellent opportunity for future 
research. 

Overall, the results of this analysis demonstrate that 
graduates of the Naval Academy who majored in engineering 
are satisfied with their undergraduate education; they 
believe that most of what they studied is applicable to what 
they do now; and that they were well prepared in most areas. 
Possible areas for program improvement would be to give 
students more opportunities to learn how to prepare and 
deliver professional presentations and prepare written 
reports and documents; these skills will greatly benefit 
them after graduation. Additional opportunities for students 
to participate in multi-disciplinary groups, or to work on 
group projects, will provide them with valuable experience 
that will be useful to them in their professional careers. 
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APPENDIX A: 


GRADUATE PERFORMANCE ASSESSMENT SURVEY 


USNA ABET GRADUATE SURVEY 
Division of Engineering and Weapons 
Administered Jan-Feb 2006 


Survey flow: 

1. Everyone takes the first part of the survey. 

2. Only currently active military take the ‘active military’ part of the survey. 

3. All others take the ‘non-active military’ part of the survey. 

4. Everyone takes the last part of the survey. 


The next page starts the FIRST PART of the survey. 
(Everyone takes this part of the survey.) 
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USNA ABET Graduates Survey 


Page 1 of 2 



Graduate Performance 
Assessment Survey 

Phase I: USNA Engineering Majors Graduates 



This survey is intended to collect assessment information regarding each of our engineering 
programs (majors) at USNA. The information collected via this survey will be analyzed and 
used by each department to help improve their respective programs in order to best prepare 
future graduates for their military and professional careers. 

You are asked, as a graduate of a USNA engineering program, to provide the following 
information voluntarily and anonymously. No identifying information will be linked to your 
responses. Questions regarding the survey may be directed toward Associate Professor Jennifer 
Waters . 


What is your current military duty status? 


o 

Active Duty (non-reservist) 

o 

Reservist on Active Duty (>=180 days. Recall or Mobilized) 

o 

Reservist on Active Duty (<180 days, AT, ADT, or ADSW) 

o 

Drilling Reservist (SELRES or IRR) 

o 

Retired (Active or Reserve) 

o 

Civilian 


How many years were you or have you been in active duty status since 
graduating from USNA? 

Select # of Years ^ 

Since graduating from USNA, how many years (if any) have you been in a non¬ 
active duty status? 

Select # of Years ^ 

4. What is your current or highest military rank achieved? 


o]|o-i I 

“II I 


http;//www. usna.edu/IR/surveys/abet-grads/stepl.php 


1/24/2006 
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a2 


G-3 


0-4 

o 

0-5 

') 

0-6 

o 

flag Rank 


I Continue | 


http://www.usna.edu/IR/surveys/abet-grads/stepl.php 


1/24/2006 
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The next page starts the ACTIVE MILITARY PART of the survey. 
(Active military only take this part of the survey.) 
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ABET - USNA Graduate Performance Assessment Survey 


Page 1 of 2 



Graduate Performance 
Assessment Sui'vey 

Phase I: USNA Engineering Majors Graduates 



In which community do you serve? 


o 

Aviation 

o 

Spec Ops 

o 

Spec Warfare 

o 

Subs 

o 

Surface 

o 

Restricted Line / Staff Corps: AEDO 

o 

Restricted Line / Staff Corps: AMDO 

o 

Restricted Line / Staff Corps: Chaplain Corps 

o 

Restricted Line / Staff Corps: Civil Engineer Corps 

o 

Restricted Line / Staff Corps: Engineering Duty 

o 

Restricted Line / Staff Corps: Human Resources 

o 

Restricted Line / Staff Corps: Information Professional 

o 

Restricted Line / Staff Corps: Information Warfare 

o 

Restricted Line / Staff Corps: Intelligence 

o 

Restricted Line / Staff Corps: JAG Corps 

o 

Restricted Line / Staff Corps: Medical Assignments 

o 

Restricted Line / Staff Corps: METOC 

o 

Restricted Line / Staff Corps: Public Affairs 

o 

Restricted Line / Staff Corps: Supply Corps 

o 

Marine Air 

o 

Marine Ground 

o 

Other 


Please list any military professional schools you have attended (e.g., flight 


http;//www. usna.edu/IR/surveys/abet-grads/step2.php 


1/24/2006 
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ABET - L’SNA GradmUe Perfomiance Assessment Survey 


Page 2 of2 


school, nuclear power school, TBS, etc.): 


Would you assess the nature of your current position as technical or non¬ 
technical? 


o 

Technical 

o 

N'on-technical 


Do you intend to complete a career sen ing in the military ? 


o 

IJefinitely Yes 

o 

1 ndecided 

o 

Definitely No 


I Continue ] 


http://www.usna.edu/IR/surveys/abet-grads/step2.php 


1/24/2006 
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The next page starts the NON-ACTIVE MILITARY PART of the survey. 
(Non-active military only take this part of the survey.) 
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ABET - USNA Graduate Performance Assessment Survey 
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Phase I: USNA Engineering Majors Graduates 


Graduate Performance 
Assessment Survey 



Please answer the following questions regarding your CURRENT PROFESSIONAL 
(CIVILIAN) EMPLOYMENT: 

What is your employment status? 


o 

Currently employed full-time 

o 

Currently employed part-time 

o 

Self-employed 

o 

Not currently employed, and looking 

o 

Not currently employed, and not looking 

o 

Retired full-time 


If you are currently employed full or part-time, please complete the following: 

Position Title: 

Name of Employer: 

City/State: 


Which of the following choices hest describes your employment and employer? 


o 

Engineering/Technical position in an engineering/techitically-based organization 

o 

Non-technical position in a technically-based organization 

o 

Technical position in a non-technically based organization 

o 

Non-technical position in a non-technically-based organization 


Which of the following levels best describes your current position? 

I — I 


http://www.usna.edu/IR/surveys/abet-grads/step2.php 


1/24/2006 
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Senior Executive (President. CEO. COO. etc.) 


ftpper Management (Vice President. Director, etc.) 


Senior Technical Professional 

o 

Mid-l.cvcl Manager 


Mid-lxfvel Technical Professional 

o 

1/ntry Level 


Other 


Is your current position related to your major at USNA? 


■I 

To a great e.xtent 

ol 

Somewhat 

O 

Not at all 


How many years have you held your current position? 

Select # of Years 

I Continue ] 


http://www.usna.edu/IR/surveys/abet-grads/step2.php 


1/24/2006 
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The next page starts the LAST PART of the survey. 
(Everyone takes this part of the survey.) 
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ABET - USNA Graduate Performance Assessment Survey 


Page 1 of 5 



Graduate Performance Assessment 
Survey 

Phase I: USNA Engineering Majors Graduates 



Are you willing to be listed in a database for midshipmen to contact regarding caree 
or other professional information? 


o 

No 

o 

Yes - Please give e-mail address or other contact info below. Alternately, if you prefer, youma> 
send an e-mail to: Associate Professor Jennifer Waters. iwaters®.usna.edu. 

Enter your contact information here: 


Upon completion of your engineering majors program at USNA, rate your preparation for the 
following abilities/characteristics: 



Not at all 
Prepared 

Somewhat 

Prepared 

Adequate 

Very 

Prepared 

Extreme 

Prepare 

Ability to apply knowledge of math, 
engineering, and science. 

O 

o 

o 

O 

O 

Ability to analyze and interpret data. 

o 

o 

o 

o 

o 

Abiiity to identify, formuiate, and solve 
engineering problems. 

o 

o 

o 

o 

o 

Ability to design system, component, or 
process to meet needs. 

o 

o 

c 

o 

o 

Ability to design and conduct 
experiments. 

o 

o 

o 

o 

o 

Ability to use techniques, skills, and tools 
in engineering practice. 

o 

o 

o 

o 

o 

Expectation to be current in technologies 
related to your career field. 

o 

o 

o 

o 

o 

Ability to prepare reports and 
documents. 

o 

o 

o 

o 

o 


http;//www. usna.edu/IR/surveys/abet-grads/step3.php 
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Ability to prepare and deliver 
professional presentations. 

o 

o 


° 

o 

Ability to function on niulti-disciplinani 
teams. 

o 

o 

o 

o 

o 

I 'nderstanding of ethical and 
professional responsibility. 

o 

o 

o 

o 

o 

Recognition of need to engage in life-long 
learning. 


o 

o 

o 

o 


In your eurrent position, how important are the follow ing? 



Not at all 
Important 

Somewhat 

Important 

Fairly 

Important 

VetT 

Important 

Extremely 

Important 

Not 

.Vpplicab 

■fo apply knowledge of 
math, engineering, and 
science. 

° 


o 

o 

o 

o 

To analyze and interpret 
data to solve open-ended 
problems. 

o 

o 

o 

o 

o 

o 

To identify, formulate, and 
solve engineering 
problems. 

o 

o 

o 

o 

o 

o 

■fo design system, 
component, or process to 
meet needs. 

o 

o 

o 

o 

o 

o 

To design and conduct 
experiments. 

o 

o 

o 

o 

o 

o 

To use techniques, skill.s, 
and tools in engineering 
practice. 

o 

o 

o 

o 

o 

o 

To be current in 
technologies related to 
your career field. 

o 

o 

o 

o 

o 

o 

To prepare reports and 
documents. 

o 

o 

o 

o 

o 

o 

To prepare and deliver 
professional presentatioiu. 

o 

o 

o 

o 

o 

o 

To function on multi¬ 
disciplinary teams. 

o 

o 

o 

o 

o 

o 

Ethical and professional 
responsibility. 

o 

o 

o 

o 

o 

o 

fo engage in life-long 


1 1 





http://\vww.usna.cdu'lR/sur\'cys abet-grads step3.php 
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lolololololo 


How well did your major at USNA prepare you for your current career? 


|o 

Not at all Prepared 

|o 

Somewhat Prepared 

[o 

Adequate 

[o 

Very Prepared 

|o 

Extremely Prepared 


How often do you attend professional conferences, meeting’s, symposia, etc.? 


ce per year 



OlBetvvecn two (2) and five (5) times per \ ear 


OllMore thiui five (5) times per year 


Do you hold a professional engineer's (P.E.) license? 


|o 

Yes 

|o 

I have passed the EE or Eff exam, and intend to pursue P.E. licensing 

lo 

I have passed the FE or EIT exam, but do not intend to pursue P.E. licensing 

|o 

No. but I intend to pursue P.E. licensing 

p 

No and do not intend to pursue P.E. licensing 


List the professional associations or societies to which you belong: 


List any significant technical honors, awards, or patents received: 


http: 'w\vw.usna.edu IR sur\'cys/abet-grads step3.php 


1 24 2006 
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Whether or not you have attended graduate <>chool, how well do you believeyw/r 
majors program at USNA prepared you for graduate school? 


|o 

Not at all Prepared 

|o 

Somewhat Prepared 

lo 

Adequate 

[o 

Very Prepared 


Extremely Well Prepared 


Indicate the highest degree you have attained: 


10||Ph l> degree or equivalent 


10||Master's degree or equivalent 


10|Bachelor's degree 


10|Some graduate work 

lo| 


Please specify' each additional degree - including discipline - you have attained since 
graduating from USNA and the school(s) attended. For example: Ph.D., Mechanical 
Engineering, University of Mary land; M.S., Systems Engineering, Naval 
Postgraduate School. 


Briefly list the most useful aspect(s) of yoiir engineering majorspntgram at L’lVA/I: 


http: 'w\vw.usna.edu IR sur\'cys/abet-grads step3.php 


1 24 2006 
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Briefly list the least useful aspect(s) of joiir engineering majors program at US.\A: 


Additional comments or suggestions for improvement regarding.)v>«r engineering 
majors program at tVV.V. l: 


Please describe any experiences or feedback related to your academic program not 
adequately captured elsesvhere in this survey. 


I Submit Information ] 


http://www.usna.edu/IR/surveys/abet-grads/step3.php 


1/24/2006 
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APPENDIX B: SAMPLE DESCRIPTIVE STATISTICS 


Table 19: Highest Military Rank Achieved 


Rank 

Frequency 

Percent 

0-1 

87 

8.1 

0-2 

107 

10.0 

0-3 

493 

46.2 

0-4 

268 

25.1 

0-5 

108 

10.1 

0-6 

1 

0.1 

Missing 

4 

0.4 

Total 

1068 

100.0 





Table 20: Warfare Community 


Community 

Frequency 

Percent 

Aviation 

224 

43.5 

Submarine 

Warfare 

79 

15.3 

Surface 

Warfare 

48 

9.3 

USMC 

69 

13.4 

SPECWAR / 
SPECOPS 

9 

1.7 

Civ. Eng. 

Corps / 
Eng. Duty 
Officer 

46 

8.9 

Other 

40 

7.8 
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Table 21: Job Description (Military) 


Job Description 

Frequency 

Percent 

Technical 

357 

69.3 

Non-Technical 

158 

30.7 

Total 

515 

100.0 


Table 22: Job Description (Civilian) 


Job Description 

Frequency 

Percent 

Engineering / 
Technical Position 
in Engineering / 
Technically based 
organization 

259 

48.0 

Non-technical 
position in 
Technically based 
org. 

107 

19.8 

Technical position 
in Non-Tech. org. 

49 

9.1 

Non-tech position 
in Non-tech org. 

125 

23.1 

Total 

540 

100.0 
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Table 23: Job Level (Civilian) 


Level within 
Organization 

Frequency 

Percent and 
Valid Percent 

Cumulative 

Percent 

Senior Exec. 
(President, 
CEO, COO, 

CIO, CFO) 

40 

7.4 

7.4 

Upper Mngmt 
(VP, 

Director) 

101 

18.7 

26.1 

Senior 
technical / 
Professional 

118 

21.8 

47.9 

Mid-level 

Management 

130 

24.0 

71.9 

Mid-level 
technical / 
professional 

107 

19.8 

91.7 

Entry Level 

18 

3.3 

95.0 

Other 

27 

5.0 

100.0 

Total 

541 

100.0 
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